Method for splitting one or more images of a sample into image data split according to dyes, computer program, computer-readable medium storing the program and system for performing the method

ABSTRACT

One or more images of a sample may be split into image data split according to dyes. The sample has at least two different dyes, in particular fluorescent dyes. The method for splitting the one or more images includes providing the one or more images of the sample and inputting the one or more images into a machine learning system. The method then includes generating the image data split according to dyes from the image or the images, using the machine learning system. The machine learning system removes at least one partial structure of the sample that is present in the image data split according to dyes of more than one dye from the image data of one or more dyes.

The invention relates to a method for splitting one or more images of asample into image data split according to dyes, a computer programproduct, a computer-readable medium storing the computer program productand a system for splitting one or more images of a sample into imagedata split according to dyes.

PRIOR ART

In fluorescence microscopy, samples (e.g. biological samples) are markedwith a plurality of fluorescent dyes. The images recorded by amicroscope, for example, are subsequently separated according tofluorescent dyes. As a result of overlapping excitation spectra and/oremission spectra of the fluorescent dyes, optical artefacts or elementsassigned erroneously or incorrectly to the respective fluorescent dyeusually occur in the images split according to fluorescent dyes(so-called “bleed-through” or so-called “crosstalk”). Specifically, thismeans that in the resulting images separated according to fluorescentdyes, pixels or structures from one color channel or one channel of onefluorescent dye are also visible (in an attenuated manner) in another(at least one other) color channel or channel of another fluorescentdye, i.e. elements or structures are in part assigned to the incorrectchannel or to the incorrect fluorescent dye (so-called artefacts).

By way of example, linear unmixing is used for splitting the image orimages according to fluorescent dyes. During linear unmixing, the imageor images is/are unmixed pixel by pixel, i.e. split according tofluorescent dyes. FIG. 4a and FIG. 4b show channels or images or imagedata 30′, 31′ split according to fluorescent dyes by means of linearunmixing in accordance with the prior art. In FIG. 4b , the linearstructures are optical artefacts, i.e. elements/pixels from the channelor image or of the fluorescent dye shown in FIG. 4a that are assignedincorrectly to this channel or image or this fluorescent dye.

What is disadvantageous about this, however, is that usually even afterlinear unmixing has been carried out, in the channels or images splitaccording to fluorescent dye, optical artefacts (i.e. pixels assigned tothe incorrect fluorescent dye) are present, inter glia on account ofrandom influences (e.g. noise) influencing the values in the image,and/or on account of the fact that the spectra of the fluorescent dyesused are not known exactly.

DISCLOSURE OF THE INVENTION

The present invention is based on the object of disclosing a method andrespectively a device and respectively a system by means of which animage or images of samples is/are or can be split into image dataaccording to dyes in such a way that the image data split according todyes have few to no optical artefacts or elements allocated or assignedto the incorrect dye in each case.

This object is achieved in each case by means of a method for splittingone or more images of a sample into image data split according to dyesas claimed in claim 1 and respectively a computer program product asclaimed in claim 16 and respectively a computer-readable medium storingthe computer program product as claimed in claim 17 and respectively asystem for splitting one or more images of a sample into image datasplit according to dyes as claimed in claim 18.

In particular, the object is achieved by means of a method for splittingone or more images of a sample into image data split according to dyes,wherein the sample has at least two different dyes, in particularfluorescent dyes, wherein the method comprises the following steps:providing the one or more images of the sample; inputting the one ormore images into a machine learning system; and generating the imagedata split according to dyes from the image or the images by means ofthe machine learning system, wherein the machine learning system removesat least one partial structure of the sample that is present in theimage data split according to dyes of more than one dye from the imagedata of one or more dyes.

One advantage of this is that without explicit specification orpredefinition of the mapping of the input data (images of the sample)onto the output data (image data separated according to dyes) followingtraining (e.g. with training data), the machine learning systemgenerates image data or images with few to no elements or opticalartefacts allocated to the incorrect dye (i.e. pixels or structures inimage data of one dye that bleed through into image data of another dyeor represent crosstalk there). What is also advantageous is that themachine learning system can generate image data that are superior tolinear unmixing such as is known from the prior art, since image datasplit according to dyes are output which have fewer elements or opticalartefacts allocated to the incorrect dye or incorrect assignments to therespective image data. Moreover, by means of the learning system, imagesof a multiplicity of samples of different kinds, without significantchanges in the machine learning system, can be decomposed or splitrapidly and technically simply into image data split according to dyeswith few to no elements or optical artefacts allocated to the incorrectdye. What is advantageous, moreover, is that the method does not carryout an assignment to the respective dye pixel by pixel, but ratheroperates in a structure-based manner. Structures or partial structuresof the sample can be recognized for example by means of edge recognitionor the like. If images already separated according to dye previously areinput as input data into the machine learning system, elements oroptical artefacts allocated to the incorrect dye and present in theimages input can be prevented or suppressed in the generated image databy means of the method, i.e. the generated image data contain fewerelements or optical artefacts allocated to the incorrect dye than theinput images already separated according to dye previously. If theimages input have not already been separated according to dyespreviously, i.e. the image or images has/have in each case data of aplurality of dyes, then the machine learning system outputs image datasplit according to dyes in which few to no elements or optical artefactsallocated to the incorrect dye are present. Prevention of the occurrenceof optical artefacts in the image data split according to dyes is thususually achieved.

Splitting the one or more images of the sample can mean, in particular,that proceeding from raw data or untreated images, image data splitaccording to dyes are generated as output which contain few or noelements or optical artefacts assigned to the incorrect dye, or thatproceeding from images already separated according to dyes previously,elements or optical artefacts assigned to the incorrect dye are removedfrom said images, such that image data split according to dyes areoutput which have few or no elements or optical artefacts assigned tothe incorrect dye.

In particular, in the course of splitting according to dyes, an elementis assigned correctly or assigned to the correct dye if the respectiveelement comprises the respective dye or is marked with the latter. Thepartial structures removed from the image data of one or more dyes canbe allocated to image data of one or more other dyes, such that a kindof displacement of the partial structure or of the image data of thepartial structure from image data of a first dye into image data of asecond dye is carried out. However, it is also possible that the partialstructure or the image data of the partial structure is/are removed onlyfrom image data of one or more image data, but the partial structure orthe image data of the partial structure is/are assigned to no image dataof other dyes, that is to say that no displacement of image data takesplace.

In particular, the object is also achieved by means of a computerprogram product having instructions which are readable by a processor ofa computer and which, when they are executed by the processor, cause theprocessor to carry out the method described above.

In particular, the object is also achieved by means of a system forsplitting one or more images of a sample into image data split accordingto dyes, wherein the sample has at least two different dyes, inparticular fluorescent dyes, wherein the system comprises a machinelearning system trained to carry out the following: generating the imagedata split according to dyes from one or more images of the sample inputinto the machine learning system, by means of the machine learningsystem, wherein the machine learning system removes at least one partialstructure of the sample that is present in the image data splitaccording to dyes of more than one dye from the image data of one ormore dyes.

One advantage of this is that without explicit specification orpredefinition of the mapping of the input data (images of the sample)onto the output data (image data separated according to dyes), thesystem generates image data or images with few to no elements or opticalartefacts allocated to the incorrect dye (i.e. pixels or structures inimage data of one dye that bleed through into image data of another dyeor represent crosstalk there). What is also advantageous about thesystem is that the system generates image data that are superior tolinear unmixing such as is known from the prior art, since image datasplit according to dyes are output which have fewer elements or opticalartefacts allocated to the incorrect dye or incorrect assignments to therespective image data. Moreover, by means of the system, images of amultiplicity of samples of different kinds, without significant changesin the system, can be decomposed or split rapidly and technically simplyinto image data split according to dyes with few to no elements oroptical artefacts allocated to the incorrect dye. What is advantageous,moreover, is that the system does not carry out an assignment to therespective dye pixel by pixel, but rather operates in a structure-basedmanner. If images already separated according to dye previously areinput as input data into the system, elements or optical artefactsallocated to the incorrect dye and present in the images input can beprevented or suppressed in the generated image data by means of thesystem, i.e. the generated image data contain fewer elements or opticalartefacts allocated to the incorrect dye than the input images alreadyseparated according to dye previously. If the images input have notalready been separated according to dyes previously, i.e. the image orimages has/have in each case data of a plurality of dyes, then thesystem outputs image data split according to dyes in which few to noelements or optical artefacts allocated to the incorrect dye arepresent.

In accordance with one embodiment of the method, the method furthermorecomprises the following steps: inputting reference images into themachine learning system; and comparing the image data of the referenceimages split according to dyes by means of the machine learning systemwith image data of the reference images split correctly according todyes for training the machine learning system for improved splitting ofthe one or more images of the sample into image data split according todyes. What is advantageous about this is that the machine learningsystem can be trained rapidly and technically simply without explicitspecification or predefinition of the mapping of the input data onto theoutput data. The comparing can take place in an automated orcomputerized manner, e.g. by means of the machine learning system, suchthat no manual intervention or no intervention by a human being isnecessary in particular during the comparing. The image data splitcorrectly according to dyes contains substantially no elements oroptical artefacts allocated to the incorrect dye. In the case of theimage data split correctly according to dyes, elements or opticalartefacts allocated to the incorrect dye may have been removed manuallyor by a human being, for example.

In accordance with one embodiment of the method, the machine learningsystem comprises or is a neural network, in particular a deep learningsystem and/or a convolutional neural network. What is advantageous aboutthis is that elements or optical artefacts allocated to the incorrectdye in the channels can be or are suppressed even better. Moreover, themachine learning system is constructed in a technically simple fashion.

In accordance with one embodiment of the method, the image is or theimages are subjected to linear or nonlinear unmixing and/or denoisingbefore being input into the machine learning system. As a result, in atechnically simple manner, the linear unmixing can be combined with thesplitting according to the dyes by means of the machine learning system,such that even fewer elements or artefacts allocated to the incorrectdye are present in the image data split according to the dye. As aresult of the denoising before inputting, the number of elements oroptical artefacts allocated to the incorrect dye in the split image datacan be reduced even further.

In accordance with one embodiment of the method, the removing of partialstructures of the sample is carried out on the basis of the structure ofthe sample, wherein in particular optical artefacts in the image dataare determined on the basis of identical and/or similar partialstructures in the image data of different dyes. As a result, the numberof elements or optical artefacts allocated to the incorrect dye in theimage data split according to dyes can be reduced even further, sincesplitting according to dyes does not take place pixel by pixel, rathersplitting according to dyes is carried out on the basis of the structureof the sample. The machine learning system thus takes account of thefact that the pixels of the image or images input are not (completely)independent of one another.

In accordance with one embodiment of the method, besides the one or moreimages spectra of the dyes used, a spectra database for estimatingand/or matching the spectra of the dyes present in the one or moreimages, recording settings and/or filter settings of the one or moreimages, detector settings of the one or more images and/or predictionsconcerning crosstalk between the different image data split according todyes are additionally input into the machine learning system. What isadvantageous about this is that the splitting according to dyes can becarried out even more reliably or more correctly.

In accordance with one embodiment of the method, the image data compriseoutput images, wherein in particular each output image shows in eachcase a dye of the image or the images input. One advantage of this isthat the image data can be viewed or examined particularly simply.Furthermore, if each output image shows in each case a dye of the imageor images input, the dyes can be examined or viewed in any desiredcombinations with one another.

In accordance with one embodiment of the method, the machine learningsystem generates a coefficient array for linear unmixing, wherein theone or more images is/are split according to dyes by means of the linearunmixing on the basis of the coefficient array. One advantage of this isthat images separated according to dyes are not generated directly,rather the coefficient array generated is or can be used to carry outsubsequent linear unmixing, the result of which is images or image datawith particularly few elements or optical artefacts allocated to theincorrect dye. As a result, splitting the images according to dyes andremoving partial structures can be considerably accelerated since thelinear unmixing requires particularly little computation time.

In accordance with one embodiment of the method, the machine learningsystem generates one or more difference images from the image data splitaccording to dyes, wherein images already separated according to dyespreviously minus the respective generated difference image associatedtherewith generate output images separated according to dyes. What isadvantageous about this is that a respective difference image isgenerated for the dyes, by which difference image the image respectivelyassociated with the dye has to be corrected in order to attain images orimage data with the fewest possible elements or optical artefactsallocated to the incorrect dye. Difference images can be interpretedmore intuitively by human beings during the training of the machinelearning system. Moreover, difference images are more readily acceptedby researchers since the correction of the images already splitaccording to dyes previously (e.g. images split previously by means oflinear unmixing) by the machine learning system is comprehensible tohuman beings. In addition, it has been found that the training of amachine learning system that maps onto difference images or correctionimages or outputs such images (so-called residual learning) issignificantly faster or more effective and yields better results, i.e.generates image data with fewer elements or optical artefacts allocatedto the incorrect dye.

In accordance with one embodiment of the method, the plurality of imagescomprise or are recordings of the same sample that are temporally offsetwith respect to one another and/or comprise or are recordings of thesame sample that are offset with respect to one another along an opticalaxis of a microscope by which the images were recorded. As a result, inparticular temporal developments or changes, e.g. of living samples (forexample animals, cells, etc.), can be split particularly reliably intoimage data separated according to dyes. As a result, a so-called Z-stack(i.e. a stack of images along the optical axis of the microscope) can bedecomposed by means of the machine learning system into image data splitaccording to dyes, wherein the image data have few to no elements oroptical artefacts allocated to the incorrect dye.

In accordance with one embodiment of the method, the machine learningsystem additionally determines or estimates the number of dyes used formarking the sample and/or identifies the dyes used for marking thesample. What is advantageous about this is that the number and/oridentity of the dyes used need not be input into the machine learningsystem. Moreover, further information about the sample can be determinedby the machine learning system.

In accordance with one embodiment of the method, the machine learningsystem furthermore determines for each pixel of the image data output aconfidence value indicating the probability that this pixel was assignedto the correct image data corresponding to the dye, and/or determinesthe absolute concentration of the respective dye in the respective imagedata and/or determines the number of objects in the sample. Thedetermination of the confidence value has the advantage that the qualityof the split image data can be better assessed since the degree ofcertainty of the respective assignment of the pixel to the respectiveimage data (so-called confidence) is indicated in each case. Moreover,it is conceivable to generate for the image data split according to dyesa respective confidence map, i.e. a two-dimensional map indicating theconfidence value for each pixel or each partial structure of the sample.The determination of the absolute concentration of the dyes (following acalibration) thus yields further information about the sample. Thedetermination of the number of objects yields further information aboutthe sample, which, in an automated manner, for example, can be processedfurther or be used for categorizing the images of the sample.

In accordance with one embodiment of the method, the machine learningsystem is trained by means of first training recordings, wherein thefirst training recordings comprise recordings of samples marked in eachcase only with one dye, and/or is trained by means of second trainingrecordings, wherein the second training recordings comprise combinedrecordings of image data already split correctly in each case accordingto dye. Particularly efficient and reliable training of the machinelearning system is achieved by means of the first training recordings.By means of the second training recordings, the machine learning systemcan be trained with particularly little complexity or within a veryshort time.

In accordance with one embodiment of the method, the image data splitcorrectly according to dyes comprise or are simulated image data, inparticular the image data split correctly according to dyes comprise orare simulated image data generated by means of a physical model, whereinpreferably the physical model comprises a physical model of a recordingoptical unit, and/or the reference images are generated by means of aphysical model from the image data split correctly according to dyes. Asa result, training data of the machine learning system can be generatedin large numbers. The physical model can contain in particular a modelthat simulates the excitation of the sample with different lasers and/orthe noise when a true recording or a true image of a sample arises or isgenerated. The image data split correctly according to dyes can be mixedtogether by means of a physical model and in this way the referenceimages can be generated in a technically simple manner.

In accordance with one embodiment of the method, the machine learningsystem determines spectra of the dyes. Consequently, if e.g. a spectradatabase was used, the spectra can be determined or refined moreaccurately.

The dyes can comprise or be fluorescent dyes and/or bright field dyes.

It is conceivable that if the dyes comprise or are fluorescent dyes, oneof the at least two fluorescent dyes which the sample has or with whichthe sample is marked is a natural (not added) fluorescent dye of thesample having autofluorescence, that is to say that the sample has afluorescent dye added to the sample and a natural autofluorescent dye(already originally present in the sample) or is marked with said dyes.

Splitting into image data split according to dyes can mean, inparticular, that respective image data are assigned in each case toexactly one dye or have only information or radiation or fluorescencefrom exactly one dye. It is also conceivable, however, for respectiveimage data to have information from more than one dye. In this regard,by way of example, an image of a sample can be split into two sets ofimage data, wherein a first set has information or radiation orfluorescence from two dyes, while a second set has information orradiation or fluorescence from exactly one dye.

The sample usually has a structure, which in turn has partialstructures. In this regard, a biological sample can have e.g. a cellnucleus and cell walls as structure or partial structures. The cellnucleus can be a partial structure in this example. The cell wall or thecell walls can be a further partial structure in this example. Partialstructures of the sample can be separated from one another for exampleby edges in the image or non-fluorescent regions in the image.

When removing a partial structure of the sample, wherein the partialstructure is present in the image data split according to dyes of morethan one dye, from the image data of one or more dyes, it is thuspossible for example for the cell nucleus of a cell that (is presente.g. incorrectly in image data of two dyes) to be removed from the imagedata of a first dye. Consequently, in the method, individual pixels aretypically not removed from image data of a dye, rather partialstructures of the sample, i.e. regions recognized as contiguous or asbelonging to a partial structure of the sample, are removed from theimage data of a dye. Structure-based removal, in particular, can thus becarried out.

Elements or optical artefacts assigned to the incorrect dye can be, inparticular, pixels or regions or partial structures of the sample whichare or were assigned incorrectly to image data of a dye or to a dye.This means, in particular, that pixels or regions or partial structuresthat are marked with only a first dye, for example, are also present insplit image data of a second dye generated by the machine learningsystem, said second dye being different than the first dye. Elements oroptical artefacts assigned to the incorrect dye are also referred to asbleed-through or crosstalk.

The optical axis of the microscope runs in particular along thedirection in which the recording is effected by means of the microscope.

Preferred embodiments are evident from the dependent claims. Theinvention is explained in greater detail below with reference todrawings of exemplary embodiments. In the figures:

FIG. 1 shows a schematic illustration of one embodiment of the methodaccording to the invention for splitting one or more images of a sampleinto image data split according to dyes;

FIG. 2 shows a fluorescence image of a sample before one or more imagesof a sample is/are split into image data split according to dyes inaccordance with one embodiment of the method according to the invention;

FIGS. 3a, 3b show images of the fluorescence image from FIG. 2 splitaccording to fluorescent dyes after one or more images of a samplehas/have been split into image data split according to dyes inaccordance with one embodiment of the method according to the invention;and

FIGS. 4a, 4b show channels or images split according to fluorescent dyesby means of linear unmixing in accordance with the prior art.

The same reference numerals are used in the following description foridentical parts and parts having an identical effect.

FIG. 1 shows a schematic illustration of one embodiment of the methodaccording to the invention for splitting one or more images of a sampleinto image data 30-32 split according to dyes. FIG. 2 shows afluorescence image of a sample before one or more images of a sampleis/are split into image data 30-32 split according to dyes in accordancewith one embodiment of the method according to the invention. FIGS. 3a,3b show images of the fluorescence image from FIG. 2 split according tofluorescent dyes after one or more images of a sample has/have beensplit into image data split according to dyes in accordance with oneembodiment of the method according to the invention.

An embodiment with fluorescence images and fluorescent dyes is explainedbelow. However, the indications and explanations are likewise applicableto images in general and dyes in general.

A sample is usually marked with more than one fluorescent dye, e.g. twoor three fluorescent dyes. Afterward, one fluorescence image or image 20or a plurality of fluorescence images of the marked sample is or arerecorded by means of a microscope.

The sample can be a biological sample, e.g. living or dead cells orliving or dead organisms, or a physical sample, i.e. an inanimatesample.

It is endeavored to split the fluorescence image or fluorescence imagesinto images or image data 30-32 according to fluorescent dyes asaccurately or correctly as possible, such that only fluorescence of onefluorescent dye is present in each image or each set of image data 30-32(so-called unmixing) It is also possible, however, for one set of imagedata 30-32 or one image to contain only fluorescence of one fluorescentdye, and the other set of image data 30-32 or another image to containthe fluorescence e.g. of the two remaining fluorescent dyes.

However, in hitherto known methods for splitting one image 20 or imagesinto image data 30-32 split according to dye, so-called elements oroptical artefacts assigned to the incorrect dye can occur in the imagedata 30-32 split according to fluorescent dyes. An element or opticalartefact assigned to the incorrect dye is or are pixels or a structureor partial structure of the sample which is or are assigned to a set ofimage data 30-32 or to an image which is not the set or imagecorresponding to the fluorescent dye. That is to say that after theimage 20 or the images 20 has/have been split into image data 30-32according to fluorescent dyes, pixels of (partial) structures of thesample that are marked only with a second fluorescent dye occur in theimage for a first fluorescent dye. This is also referred to as so-calledbleed-through between the different images/image data 30-32 or asso-called crosstalk between the images/image data 30-32.

The elements or optical artefacts assigned to the incorrect dye mayarise or be brought about e.g. as a result of random influences (such asthe noise of the input data or of the fluorescence images) and/or as aresult of the inexact knowledge about the spectra of the fluorescentdyes used.

A machine learning system 10 or a system for machine learning is usedfor this splitting into image data 30-32 or sets of image data 30-32.The system is configured for splitting one or more images of a sampleinto image data 30-32 split according to dyes and comprises a machinelearning system 10.

The machine learning system 10 can comprise a sparse dictionary learningsystem and/or an autoencoder.

The machine learning system 10 can comprise or be e.g. a neural network.In particular, the machine learning system 10 can comprise or be a deeplearning system. Moreover, it is possible for the machine learningsystem 10 to comprise or be a convolutional fluorescent dye or a deepconvolutional neural network.

Moreover, the machine learning system 10 can comprise or be a multilayerperceptron, in which besides the output layer there is also at least onefurther layer of hidden neurons (so-called hidden layer). All theneurons of one layer are completely linked to the neurons of the nextlayer with forward propagation (so-called feedforward network).

Furthermore, it is also conceivable for the machine learning system 10to comprise or consist of one or more so-called generative adversarialnetworks.

The machine learning system 10 is trained to the effect of carrying outthe splitting of images, in particular fluorescence images, into imagedata 30-32 split according to fluorescent dyes and removing elements oroptical artefacts assigned to the incorrect dye or preventing orsuppressing the presence of elements or optical artefacts assigned tothe incorrect dye. The structure or structures of the samplerespectively imaged is or are taken into account for this purpose. Humanbeings can recognize the elements or optical artefacts assigned to theincorrect dye usually on the basis of the context information, e.g. ifthe identical or very similar structures or partial structures occurwith varying intensity or concentration (e.g. brightness) in more thanone channel, e.g. in relation to the wavelength of channels adjacent toone another.

By means of corresponding training, the machine learning system 10 istrained to carry out on the basis of said structures the splitting ofone or more fluorescence images 20 into image data 30-32 split orseparated according to fluorescent dyes, in order to generate image data30-32 or images with the fewest possible or no elements or artefactsassigned to the incorrect dye (i.e. incorrectly assigned pixels orstructures). The machine learning system 10 recognizes at least onepartial structure of the sample that is present in the image data 30-32split according to dyes of more than one dye as an element or opticalartefact assigned to the incorrect dye and removes the respectivepartial structure or the element or optical artefact assigned to theincorrect dye from the image data 30-32 of one or more dyes or preventsthe occurrence or presence of the element or optical artefact assignedto the incorrect dye in the image data 30-32 output. Consequently, theoccurrence or presence of elements or optical artefacts assigned to theincorrect dye in the image data 30-32 is prevented or reduced and thenumber of elements or optical artefacts assigned to the incorrect dye inthe image data 30-32 generated thus decreases.

The input data input into the machine learning system 10 can comprise astack or cluster of fluorescence images that includes as many channelsas desired.

The fluorescence images can be raw data, i.e. unprocessed fluorescenceimages, or else be present as already stored/computed fluorescenceimages, i.e. as already processed fluorescence images. The fluorescenceimages input into the machine learning system 10 as input can besingle-track, multi-track or lambda stack recordings. In the case ofmulti-track recordings, a plurality of recordings or images 20 of thesame sample are created with different settings of the laser and/or ofthe detector. A lambda stack comprises a stack or set of images whicheach comprise a specific wavelength range. The wavelength ranges of alambda stack are usually disjoint with respect to one another anddirectly adjoin one another.

In the case of unprocessed fluorescence images, the number of channelsis dependent on the recording mode of the images 20. In the case ofalready processed or computed data, the number of channels is dependenton the processing or computation of the images 20 or recordings. By wayof example, it is possible for the fluorescence images or the input dataof the machine learning system 10 already to be the result of linearunmixing, such as is known from the prior art. The number of elements oroptical artefacts which are assigned to the incorrect dye and which arestill present in the image data 30-32 or images after the linearunmixing is reduced by means of the machine learning system 10 andrespectively the method according to the invention. If the input dataare channels already separated according to dyes by means of linearunmixing, then the number of channels of the input data and/or outputdata of the method can be equal to the number of fluorescent dyes usedfor marking the sample.

The number of fluorescence images used as input of the machine learningsystem 10 can vary depending on the type of fluorescence images. Thefluorescence images can be unprocessed fluorescence images, wherein thefluorescence images are created by means of the use of a detector array(in this case, the number of fluorescence images is equal to the numberof detector windows or equal to the number of channels of the detector).Alternatively, the fluorescence images can be lambda stack recordings(in this case, the number of fluorescence images is equal to the numberof defined bands of the lambda stack). As a further alternative, thefluorescence images can be recordings with freely defined bandwidths inthe multi- or single-track mode (in this case, the number offluorescence images is typically equal to the number of dyes used).Moreover, it is conceivable for a single fluorescence image to be usedas input of the machine learning system 10.

Besides the fluorescence images, the input data of the machine learningsystem 10 can contain the following further information that is takeninto account by the machine learning system 10:

-   -   the spectra of the fluorescent dyes used that are present in the        fluorescence image 20, and/or    -   a spectra database of the fluorescent dyes used, which can serve        for estimating and coordinating/matching the spectra present,        and/or    -   settings with regard to the recording of the fluorescence        images, such as e.g. laser settings, detector settings, filter        settings, smart setup settings and/or predictions concerning        crosstalk or concerning elements or optical artefacts assigned        to the incorrect dye.

The output data of the method and respectively of the machine learningsystem 10 usually contain image data 30-32 or images split according tofluorescent dyes. The image data 30-32 can themselves be images. It isalso conceivable, however, for the method and respectively the machinelearning system 10 to output only data or image data 30-32 by means ofwhich images split according to fluorescent dyes can be calculated orcreated. For example, the data or image data can comprise or be acoefficient array that is used for subsequent linear unmixing forsplitting the fluorescence image or fluorescence images according todyes. This means that the machine learning system 10 calculates ordetermines the coefficients for the linear unmixing in such a way thatthe number of elements or optical artefacts assigned to the incorrectdye in the images split according to fluorescent dyes after linearunmixing using these coefficients is small.

Moreover, it is possible for the method and respectively the machinelearning system 10 to output so-called difference images and/or to betrained thereon. If the difference images are subtracted in each casefrom the corresponding images or channels split according to fluorescentdyes by means of hitherto known methods, this results in images or imagedata 30-32 which contain fewer or even no longer any elements or opticalartefacts assigned to the incorrect dye. This means that for each inputchannel or each fluorescence image that is input into the machinelearning system 10, the machine learning system 10 generates acorresponding difference image, wherein as a result of the subtractionof the difference image from the corresponding fluorescence image,fluorescence images (output images) are generated which contain or havefewer to no elements or optical artefacts assigned to the incorrect dye.

The number of channels that are output by the method and respectivelythe machine learning system 10 can be any desired number. In thisregard, it is conceivable for the images or image data 30-32 not only tobe split according to fluorescent dyes, but additionally also to besplit according to different regions or objects of the fluorescenceimages. Consequently, in the case of three fluorescent dyes and threeobjects, for example, there are nine channels or nine sets of image data30-32 as output of the method and respectively the machine learningsystem 10. Moreover, it is possible for the fluorescence image to besplit into two sets of image data 30-32 in regard to the threefluorescent dyes used (namely firstly one fluorescent dye in one set ofimage data or in one image and two fluorescent dyes in the other set ofimage data or in another image), with the result that in the case ofthree objects six output channels or images (=2*3) arise.

Preferably, the number of channels output by the method and respectivelythe machine learning system 10 corresponds to the number of fluorescentdyes used for marking the sample. Moreover, it is possible for thenumber of channels to correspond to the number of fluorescence imagesthat are used as input.

In the image data 30-32 split according to dyes, in each case theabsolute concentration of the respective fluorescent dye can bedetermined and displayed or output if a calibration was carried out,i.e. if there was a determination of which brightness value respectivelycorresponds to which concentration.

Furthermore, additional information can be output by the method andrespectively the machine learning system 10 in addition to the imagedata 30-32 or channels. This may be for example information that may beuseful for further processing of the channels or image data 30-32 or forrepresentation purposes for the observer of the channels or image data30-32 output. In particular, the additional information can comprise orbe the selected spectra of the fluorescent dyes used (if a spectradatabase was part of the input), the estimated spectra of thefluorescent dyes used (if the spectra of the fluorescent dyes were notpart of the input) and/or the refined or more precisely determinedspectra of the fluorescent dyes (if the spectra and/or a spectradatabase were/was part of the input).

If a spectra database is present or was part of the input, theidentifications or names of the fluorescent dyes used can additionallybe output. A spectra database indicates the respective spectra orwavelengths of the different fluorescent dyes.

If the concentrations of the fluorescent dyes are estimated by themethod and respectively the machine learning system 10, the confidencefor each pixel or each partial structure can be output or indicated bythe method and respectively the machine learning system 10. Theconfidence indicates the probability with which the respective pixel wascorrectly assigned to the respective fluorescent dye.

Moreover, it is possible for the machine learning system 10 andrespectively the method to determine and output the type, size, number,etc. of the objects marked with the fluorescent dyes in the sample. Byway of example, the number of marked cells can be determined and outputby the method and respectively the machine learning system 10.

Splitting the fluorescence images 20 into sets of image data 30-32 orchannels by means of the machine learning system 10 can take place onthe basis of the unprocessed fluorescence images and/or on the basis ofchannels or images that have already been processed or have already beenseparated according to fluorescent dyes.

Firstly, an explanation is given below for improving fluorescence imagesthat have already been processed by methods according to the prior art.

The fluorescence images 20 that serve as input for the method andrespectively the machine learning system 10 can be fluorescence images(e.g, individual images and/or detector stack images) that have alreadybeen denoised by means of a denoising algorithm.

It is possible for the image data that are output by the machinelearning system to be at least partly denoised by means of the machinelearning system.

Moreover, it is possible that if the spectra of the fluorescent dyes arealready present (from a database or on the basis of an estimation), thespectra of the fluorescent dyes are determined more accurately, that isto say that the spectra or the channels of the spectra are approximatedto the actual values of the spectra or limits of the differentfluorescent dyes and/or are adapted to the present sample. If thespectra of the fluorescent dyes used are not present, then it ispossible to determine the spectra of the fluorescent dyes directly onthe basis of the fluorescence images that influence the method andrespectively the machine learning system 10 as input. For this purpose,the machine learning system 10 is trained in such a way that it learns amapping of the input data or fluorescence images onto the spectra.

After linear or nonlinear unmixing has been carried out, elements oroptical artefacts assigned to the incorrect dye (so-called bleed-throughof one channel or of image data 30-32 of one fluorescent dye into imagedata 30-32 of another fluorescent dye) can be rectified or eliminated bymeans of the method and respectively the machine learning system 10 onthe basis of the structures present in the fluorescence image(s). Theadvantage of this structure-based analysis or processing is, inter alia,that it is possible easily to ascertain bleed-through on the basis ofidentical structures at spatially the same location in channels—normallydirectly adjacent to one another—of the output or image data 30-32. Whatis crucial in this case is not the kind or type of structure per se, butrather the position and manifestation in the different channels or imagedata 30-32. This relationship can be learned by means of machinelearning on the basis of a representative training sample, wherein thereare various possibilities for mapping input data onto output data.

In the case of a first possibility, the input data or fluorescenceimages are analyzed or determined by means of a so-called patch-basedmethod, wherein in this kind of analysis an image is analyzed on thebasis of image segments of the image or is processed image segment byimage segment. In this case, the input data or images resulting from thelinear or nonlinear unmixing are split into patches by the machinelearning system 10. In this case, the context available to the machinelearning system 10, that is to say the visible structures, is limited bythe size of the patch. Each input patch of the machine learning system10 is mapped onto an output patch, wherein a bleed-through between thechannels or image data 30-32 of the different dyes is suppressed orprevented by the machine learning system 10. Finally, the output patchesare combined to form an overall image or a channel. This means thatregions or patches of the overall fluorescence image are input to themachine learning system 10 successively and, after the splittingaccording to dyes, the regions or patches are combined again to form anoverall fluorescence image.

A second possibility comprises a so-called image-to-imagetransformation_(;) wherein the machine learning system 10 learns amapping of the overall fluorescence image onto overall images or imagedata 30-32 of the overall image or is trained thereon. The procedure isfor the most part identical to the patch-based method described above,but the use of the overall image means that a significantly largercontext can be taken into account by the machine learning system 10. Themachine learning system 10 outputs channels or images or image data30-32 of the overall image that are split according to dyes.

Furthermore, it may be advantageous for the machine learning system 10not to be trained on outputting the images or image data 30-32themselves split correctly according to dyes, but rather to be trainedon a correction or on a difference between the image data 30-32 alreadypresent and the image data 30-32 rectified vis-á-vis elements or opticalartefacts assigned to the incorrect dye, and to output such image data30-32. This means, for example, that for each channel of the output oreach set of image data or images of linear unmixing in accordance withthe prior art_(;) by means of the machine learning system 10, adifference image is estimated or generated by which the respectivechannel of the output of the linear unmixing has to be corrected inorder to obtain image data 30-32 in which elements or optical artefactsassigned to the incorrect dye have been removed or are no longer presentor no longer occur. The difference image indicates in each case thedifference between the output of the linear unmixing and the image data30-32 or images rectified by the machine learning system 10 vis-a-viselements or optical artefacts assigned to the incorrect dye. Theadvantage of this is that such difference images can be interpreted moreintuitively by human beings during the training of the machine learningsystem 10 and elements or optical artefacts assigned to the incorrectdye can thus be recognized more easily. Moreover, the correction by themachine learning system 10 is more easily comprehensible to humanbeings, with the result that this machine learning system has a higheracceptance. Moreover, it has been found that training a machine learningsystem 10 on difference images or to generate difference images(so-called residual learning) is technically very simple and yields verygood results, that is to say that by this means the number of elementsor optical artefacts assigned to the incorrect dye is greatly reduced,possibly even reduced to a greater extent than in the case of trainingthe machine learning system 10 to directly output image data 30-32 withfew elements or optical artefacts assigned to the incorrect dye.

Improvement of unprocessed fluorescence images by means of the method ormachine learning will now be described below.

In this case, the splitting into image data 30-32 ordered according tofluorescent dye is learned completely from the input data. In this case,the machine learning system 10 is trained in such a way that mapping iseffected directly from the input stack or from the fluorescence imagesonto the output stack or the image data 30-32 or images. A prerequisitein this case is a sufficiently large database for training the machinelearning system 10, wherein the database covers substantially allexpected variations or possibilities. This is achieved by means of asimulation in which a multiplicity of recordings in which the sample wasmarked or stained in each case only with one dye are mixed or added toform an input stack or to form an individual fluorescence image and thisis used in each case as input for the machine learning system 10. Therecordings in which the sample was marked or stained in each case onlywith one dye are used as reference images that are intended to be outputas image data 30-32 by the machine learning system 10. It is alsopossible to simulate different output data or channels such as, forexample, single- and multi-track data or lambda stacks as input data ofthe machine learning system 10.

Moreover, the input data or the input stack can be rendered artificiallynoisy in order to approximate the input data to true fluorescence imagesor true recording conditions to an even greater degree. Precisely in thecase of single-track recordings or lambda stacks there is often a lowersignal-to-noise ratio, i.e. a high degree of noise. Those individualimages which were mixed or added together to generate the input data orthe input stack serve as ground-truth data, i.e. as output to be strivenfor of the machine learning system 10 being trained. A mapping fromimage stack to image stack is carried out in this case.

A further possibility besides improving fluorescence images alreadyprocessed by methods according to prior art and improving unprocessedfluorescence images by means of the method or machine learning is forthe unmixing, i.e. the splitting of the image(s) input into the machinelearning system 10 according to dyes, to be implicitly learned, i.e, allrequired parameters that are estimated or predefined in traditionallinear unmixing can be projected or determined with the aid of themachine learning system 10.

In this regard, for example, the coefficients (A) for solving theequation system of linear unmixing, the number of spectra, the spectrathemselves or a combination thereof can be output by the machinelearning system 10. A mapping of an input stack or of a plurality offluorescence images onto an intermediate representation is generallycarried out in this case, i.e. image data 30-32 or data for splittingthe fluorescence images into channels are generated by the machinelearning system 10. By means of the data and e.g. the linear unmixing,the fluorescence images input into the machine learning system 10 can besplit into images or image data 30-32 split or separated according todyes in such a way that said images or image data have few to noelements or optical artefacts assigned to the incorrect dye.Furthermore, it is also possible here to input additional informationinto the machine learning system 10, such as, for example, informationabout the spectra of the dyes used,

It is also possible for the machine learning system 10 to estimate ordetermine additional information (designation of the dyes used, numberof dyes used) about the spectra and/or on the basis of the structuresoccurring in the fluorescence image that is input into the machinelearning system 10. In this regard, by way of example, cell nucleusstructures in a biological sample indicate the frequently usedfluorescent dye DAR (4′,6-diamidino-2-phenylindole), such that when cellnucleus structures are recognized, the machine learning system 10 canassume that this fluorescent dye was used.

Moreover, fluorescence images in the form of a Z-stack can be input intothe machine learning system 10, i.e. fluorescence images that wererecorded from the same sample in the manner offset with respect to oneanother along the optical axis of the microscope, and these images aresplit into image data 30-32 or images in a manner separated according todyes by means of the method and respectively the machine learning system10. In this case, the structure of the sample and/or similar oridentical structures in the channels or images of the output can berecognized by the machine learning system 10 and, as a result, elementsor optical artefacts assigned to the incorrect dye can be removed or theoccurrence of elements or optical artefacts assigned to the incorrectdye can be prevented.

Furthermore, recordings that are temporally offset with respect to oneanother can be used as fluorescence images or input of the method andrespectively the machine learning system 10. In this case, the structureof the sample and/or similar or identical structures in the channels ofthe output can be recognized by the machine learning system 10 and, as aresult, artefacts can be removed or the occurrence of elements oroptical artefacts assigned to the incorrect dye can be prevented orreduced.

The machine learning system 10 can in each case be adapted to the sampleor kind of sample (e.g. biological sample or physical sample, etc.) orbe specifically trained thereon. It is also conceivable, however, forthe machine learning system 10 not to be actively changed, independentlyof the sample.

The channels or sets of image data 30-32 that are output by the machinelearning system 10 can be equal to the number of fluorescence images ofthe input or input channels. The number of output channels of themachine learning system 10 can also correspond to the number of objectswhich the sample has.

The differentiation as to whether a specific element or a structure or apixel in the fluorescence image is or is not an element or opticalartefact assigned to the incorrect dye is based on structure informationor recognized structures or partial structures of the sample in thefluorescence image, that is to say that the entire fluorescence image ascontext influences the splitting into image data 30-32 or imagesaccording to fluorescent dye.

The machine learning system 10 can be present or implemented on acommercially available computer. Moreover, it is conceivable for themachine learning system 10 to be implemented on a computer designedspecifically for the machine learning system 10. In particular, theprocessors of one graphics card or on a plurality of graphics cards canbe used for the required calculations of the machine learning system 10and respectively of the method.

The method for training the machine learning system 10 can be carriedout for example as follows. A fluorescence image is input to the machinelearning system 10. The machine learning system 10 splits thefluorescence image into image data 30-32 or images ordered according tofluorescent dyes. Moreover, a human being identifies or has identifiedand removed in the respective channels the structures or regionsrepresenting elements or optical artefacts assigned to the incorrect dye(so-called bleed-through), i.e. the structures or regions that wereassigned incorrectly to the respective image or channel. These imagedata 30-32 or images freed of elements or optical artefacts assigned tothe incorrect dye are input to the machine learning system 10 as targetoutput or desired output. The machine learning system 10 learnstherefrom to avoid in future such elements or optical artefacts assignedto the incorrect dye. The machine learning system 10 undergoessupervised learning in this way.

A further possibility for training the machine learning system 10 is touse as training data images in which in each case the same sample orsamples that are identical or similar to one another is or are marked ineach case only with one fluorescent dye. These images are mixed or addedtogether and the result is used as fluorescence image or fluorescenceimages as input of the machine learning system 10. Since the originalimages separated according to fluorescent dye are present, the machinelearning system 10, without supervision and human assistance, can itselfrecognize which structures or regions or pixels of the result or of theoutput image data 30-32 of the machine learning system 10 represent anelement or optical artefact assigned to the incorrect dye and which donot. As a result, the machine learning system 10 can be trained veryrapidly.

The fluorescence images can also be images that were recorded by meansof two-photon fluorescence microscopy. So-called second and thirdharmonic generation (i.e. frequency doubling and/or frequency tripling)is also taken into account in this case.

The machine learning system 10 can comprise a neural network, inparticular a so-called deep neural network. The neural network comprisesfor example more than three layers, such that more than one so-calledhidden layer is present.

The machine learning system 10 can comprise or be a convolutional neuralnetwork (CNN). In this case, the machine learning system 10 comprises aconvolutional layer, which consists of a two-dimensional orthree-dimensional matrix consisting of pixels. The activity of eachneuron of this plane is calculated by means of a discrete convolution.The convolutional layer is followed by a so-called pooling layer, inwhich excess information is discarded. This can subsequently be followedagain by a convolutional layer and thereafter a pooling layer, etc.Finally, the last pooling layer is followed by a fully-connected layer.The neurons of the fully-connected layer usually correspond to thenumber of channels into which the fluorescence image is intended to besplit.

Linear unmixing is described for example in the article “Clearing up thesignal: spectral imaging and linear unmixing in fluorescencemicroscopy”, which was published in the journal “Methods in MolecularBiology” in 2014, 1075:129-48 (doi: 10.1007/978-1-60761-847-8_5).

In linear unmixing, it is assumed e.g. that a pixel is categorized aslinearly mixed if the measured spectrum (S(λ)) corresponds to the weight(A) of each individual fluorescent dye reference spectrum (R(λ)):

S(λ)=A1·R1(λ)+A2·R2(λ)+A3·R3(λ) . . . Ai·Ri(λ)

This can be expressed more generally as:

S(λ)=ΣAi·Ri(λ) or S=A·R

In these equations, the signal in each pixel (S) is measured during therecording of the lambda stack and the reference spectrum of the knownfluorescent dye is measured usually independently of one another insamples that are marked in each case only with one dye under identicalinstrument settings. Using algebra, the contributions of the differentdyes (Ai) can be determined by calculating their contribution to eachpoint in the measured spectrum.

By way of example, this can be determined by minimizing the squarebetween the measured and the calculated spectra by applying thefollowing differential equations:

[∂Σj {S(λj)-ΣAi·Ri(λj)}2]/∂Ai=0

In this equation, j represents the number of detection channels and icorresponds to the number of dyes.

It is also possible for the convolutional neural network (CNN) to co pse or be a non-rejuvenating network.

LIST OF REFERENCE NUMERALS

10 machine learning system

20 image of the sample

30, 31, 32, 30′, 31′ image data split according to dyes

1. A method for splitting one or more images of a sample into image datasplit according to dyes, wherein the sample has at least two differentdyes, comprising: providing the one or more images of the sample;inputting the one or more images into a machine learning system; andgenerating the image data split according to dyes from the one or moreimages with the machine learning system, wherein the machine learningsystem removes at least one partial structure of the sample that ispresent in the image data split according to dyes of more than one dyefrom the image data of one or more dyes.
 2. The method as claimed inclaim 1, furthermore comprising: inputting reference images into themachine learning system; and comparing the image data of the referenceimages split according to dyes with the machine learning system withimage data of the reference images split correctly according to dyes fortraining the machine learning system for improved splitting of the oneor more images of the sample into image data split according to dyes. 3.The method as claimed in claim 1, wherein the machine learning systemcomprises or is a neural network.
 4. The method as claimed in claim 1,wherein the image is, or the images are, subjected to linear ornonlinear unmixing and/or denoising before being input into the machinelearning system.
 5. The method as claimed in claim 1, wherein: theremoving of the at least one partial structure of the sample is carriedout based on a structure of the sample, wherein optical artefacts in theimage data are determined based on identical and/or similar partialstructures in image data of the at least two different dyes.
 6. Themethod as claimed in claim 1, further comprising: besides the one ormore images, inputting, into the machine learning system, spectra of theat least two different dyes, a spectra database for estimating and/ormatching the spectra of the dyes present in the one or more images,recording settings and/or filter settings of the one or more images,detector settings of the one or more images and/or predictionsconcerning crosstalk between the different image data split according todyes.
 7. The method as claimed in claim 1, wherein the image datacomprise output images, wherein each output image shows in each case adye of the image or the images input.
 8. The method as claimed in claim1, wherein the machine learning system generates a coefficient array forlinear unmixing, wherein the one or more images is/are split accordingto dyes by linear unmixing based on the coefficient array.
 9. The methodas claimed in claim 1, wherein the machine learning system generates oneor more difference images from the image data split according to dyes,wherein images already separated according to dyes previously, minus therespective generated difference image associated therewith, generateoutput images separated according to dyes.
 10. The method as claimed inclaim 1, wherein the one or more images comprise or are recordings ofthe same sample that are temporally offset with respect to one another,and/or the one or more images comprise or are recordings of the samesample that are offset with respect to one another along an optical axisof a microscope by which the images were recorded.
 11. The method asclaimed in claim 1, wherein the machine learning system additionallydetermines or estimates a number of the at least two different dyesand/or identifies the at least two different dyes.
 12. The method asclaimed in claim 1, wherein the machine learning system furthermore:determines for each pixel of the image data output a confidence valueindicating a probability that the pixel was assigned to correct imagedata corresponding to the dye, and/or determines an absoluteconcentration of the respective dye in the respective image data and/ordetermines a number of objects in the sample.
 13. The method as claimedin claim 1, wherein the machine learning system is trained by firsttraining recordings, wherein the first training recordings compriserecordings of samples marked in each case only with one dye, and/or themachine learning system is trained by second training recordings,wherein the second training recordings comprise combined recordings ofimage data already split correctly in each case according to dye. 14.The method as claimed in claim 2, wherein the image data of thereference images split correctly according to dyes comprise or aresimulated image data generated with a physical model, and/or thereference images are generated with a physical model from the image datasplit correctly according to dyes.
 15. The method as claimed in claim 1,wherein the machine learning system determines spectra of the at leasttwo different dyes.
 16. A computer program product having instructionswhich are readable by a processor of a computer and which, when they areexecuted by the processor, cause the processor to carry out the methodas claimed in claim
 1. 17. A computer-readable medium storing thecomputer program product as claimed in claim
 16. 18. A system forsplitting one or more images of a sample into image data split accordingto dyes, wherein the sample has at least two different dyes, wherein thesystem comprises a machine learning system trained, to carry out amethod comprising: generating the image data split according to dyesfrom the one or more images of the sample input into the machinelearning system with the machine learning system, wherein the machinelearning system removes at least one partial structure of the samplethat is present in the image data split according to dyes of more thanone dye from the image data of one or more dyes.
 19. The method asclaimed in claim 1, wherein the at least two different dyes comprise atleast one fluorescent dye, and the neural network is a deep learningsystem and/or a convolutional neural network.
 20. An imaging method,comprising: marking a sample with at least a first dye and a second dye,wherein the second dye is different from the first dye, obtaining, witha microscope, at least one initial image of the sample, providing asignal based on the initial image to a machine learning system,generating, with the machine learning system, at least one first outputimage from the at least one initial image, wherein the first outputimage shows the first dye and is substantially devoid of opticalartefacts of the second dye, and generating, with the machine learningsystem, at least one second output image from the at least one initialimage, wherein the second output image shows the second dye and issubstantially devoid of optical artefacts of the first dye, whereingenerating the first output image comprises removing an optical artefactof the second dye with the machine learning system based on image dataof the first and second dyes, and/or wherein generating the secondoutput image comprises removing an optical artefact of the first dyewith the machine learning system based on image data of the first andsecond dyes.